Teaching Dimension and the Complexity of Active Learning

We study the label complexity of pool-based active learning in the PAC model with noise. Taking inspiration from extant literature on Exact learning with membership queries, we derive upper and lower bounds on the label complexity in terms of generalizations of extended teaching dimension. Among the contributions of this work is the first nontrivial general upper bound on label complexity in the presence of persistent classification noise. 1 Overview of Main Results In supervised machine learning, it is becoming increasingly apparent that welldesigned interactive learning algorithms can provide valuable improvements over passive algorithms in learning performance while reducing the amount of effort required of a human annotator. In particular, there is presently much interest in the pool-based active learning setting, in which a learner can request the label of any example in a large pool of unlabeled examples. In this case, one crucial quantity is the number of label requests required by a learning algorithm: the label complexity. This quantity is sometimes significantly smaller than the sample complexity of passive learning. A thorough theoretical understanding of these improvements seems essential to fully exploit the potential of active learning. In particular, active learning is formalized in the PAC model as follows. The pool of m unlabeled examples are sampled i.i.d. according to some distribution D. A binary label is assigned to each example by a (possibly randomized) oracle, but is hidden from the learner unless it requests the label. The error rate of a classifier h is defined as the probability of h disagreeing with the oracle on a fresh example X ∼ D. A learning algorithm outputs a classifier ĥ from a concept space C, and we refer to the infimum error rate over classifiers in C as the noise rate, denoted ν. For ǫ, δ, η ∈ (0, 1), we define the label complexity, denoted #LQ(C,D, ǫ, δ, η), as the smallest number q such that there is an algorithm that outputs a classifier ĥ ∈ C, and for sufficiently large m, for any oracle with ν ≤ η, with probability at least 1 − δ over the sample and internal randomness, the algorithm makes at most q label requests and ĥ has error rate at most ν + ǫ. 1 Alternatively, if we know q ahead of time, we can have the algorithm halt if it ever tries to make more than q queries. The analysis is nearly identical in either case. The careful reader will note that this definition does not require the algorithm to be successful if ν > η, distinguishing this from the fully agnostic setting [1]; we discuss possible methods to bridge this gap in later sections. Kulkarni [2] has shown that if there is no noise, and one is allowed arbitrary binary valued queries, then O (logN(ǫ)) ≤ O (